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Reducing h^lo-like effects in motion-compensated interpolation 



The invention relates to a motion-compensated interpolation of a data-signal, 
which data-signal comprises successive images wherein each image comprises groups of 
pixels, in which motion vectors are generated, each motion vector corresponding to a group 
of pixels of one image, between a group of pixels of said one image and a second group of 
pixels of another image in the data-signal, and interpolated results are obtained as a function 
of these motion vectors. Furthermore, the invention relates to a picture signal display 
apparatus comprising means for receiving a data-signal, and a device for motion- 
compensated interpolation of this data- signal. 

US-A-5,777,682 describes that at least two motion vectors are calculated for 
each group of pixels of an image. Then, for each motion vector an interpolated result is 
calculated. Finally, the thus obtained interpolated results are averaged which finally yields an 
interpolated image. A disadvantage of that method is that the area of the halo-effect in the 
interpolated images is increased. The halo-effect is due to impairments (blurring and/or 
magnifying lens effect) mainly occurring at the boundaries of moving objects, caused by 
incorrect motion vectors and/or an incorrect interpolation strategy (not taking the occlusion 
problem into account) in the motion compensated interpolated images of the data-signal. For 
an observer of the images the halo-effect is an annoying effect. 

It is an object of the invention to provide a motion-compensated interpolation 
that reduces the area of the halo-effect in the interpolated images of the data-signal. The 
invention therefore provides a motion-compensated interpolation that further comprises: 

estimating the reliability of each motion vector corresponding to a particular 
group of pixels; 

calculating weights as a function of the reliability of the motion vectors; and 
generating an interpolated luminous intensity of a group of pixels for an 

interpolated image by calculating, on the basis of these weights, a weighted average of the 

interpolated results. 

Advantageous embodiments are defined in the dependent claims. 
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The reliability of a motion vector can be a function of the accuracy of the 
motion vector. The accuracy of the motion vector is determined by the difference of a 
predicted luminous intensity on the basis of the motion vector, wherein the motion vector is 
assigned to a first group of pixels of a first image, for a second group of pixels in a second 
image and the actual luminous intensity of the second group of pixels of the second image. 
The reliability of a motion vector can also be a function of the relative frequency of 
occurrence of the motion vector in the neighboring groups of pixels of the first group of 
pixels. Generally, there will be eight neighboring groups of pixels in the first image of the 
first group of pixels. The reliability of the motion vector can also be a function of both the 
accuracy of the motion vector and its relative occurrence. 

For each interpolation result a weight is calculated as a function of the 
reliability of the motion vector which yielded the interpolation result. Since each group of 
pixels is assigned a multiple of motion vectors this leads a multiple of interpolation results 
and corresponding weights per group of pixels. 

Eventually, the interpolated luminous intensity of a group of pixels is 
calculated as a weighted average of the interpolation results on the basis of the reliability of 
the interpolation results. This approach yields some important advantages. A first example of 
this is the situation wherein among the multiple of motion vectors just one motion vector is 
the right one. Then, the interpolation result, which corresponds to this motion vector, is 
assigned a relative great weight such that this interpolation result dominates the interpolated 
luminous intensity of the group of pixels. As a result a high accuracy of the interpolated 
intensity of the group of pixels is obtained and fine details in the picture can be shown 
without the risk of the halo-effect to appear. A second example of an advantage of the 
approach according to the invention is the situation wherein neither of the motion vectors is 
correct. In this situation the multiple of motion vectors will show a great diversity and none 
of the motion vectors will be assigned a heavy weight. The effect is that the interpolated 
intensity of the group of pixels will be a smooth average of the interpolated results. In this 
case no fine details are showed, the picture will be blurred. However, for an observer a 
blurred picture is more attractive than wrongly flickering details. So, in either of the two 
examples the halo-effect is reduced. 

An embodiment of the method of the invention is characterized in that the 
interpolated luminous intensity of a group of pixels is calculated according to: 
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I k * A U ) = (£ m =l,..,M {W k m (x ) * i k+A m (* )}) / Xm=l,..M { W k m (3c )}, (I) 

wherein I k+A (x ) is the interpolated luminous intensity of the group of pixels of the 
interpolated image F k+A , wherein the real value A defines the place of the inteipolated image 
5 F k+A in the image sequence F n , n=l ,2,...,k,k+l,...,N. Generally A is a real value in the interval 
[0,1]. Furthermore, in (I), the location of the group of pixels in the image is defined with the 
integer two-dimensional vector x, Em=i,..,M {• } is a summation from 1 to M over its argument 
{.} and w k m (* ) is a weight corresponding to the m* interpolation result i k+A m (ic ). The m* 
interpolation result i k+A m ( x ) can for example be calculated according to the following 
10 interpolation strategy: 

i k+A m( x ) = medianf (I k ( round { x - A * D* (x ) } ) , 

a k (x)+I k+1 U))/2) . (n> 
(I k+1 (round {x + (1 - A) * D k m {x)} )}, 

15 

wherein medianf.} is a function which gives the median value of its input arguments and 
roundfj is a function which gives the nearest integer value to each component of its input 
argument, and wherein I k (* ) is a luminous intensity of a group of pixels at location x in the 
image F k and wherein D* (x) is the m* two-dimensional integer motion vector of the M 

20 motion vectors which correspond to the group of pixels at location x in the image F k , which 
m* motion vector is normalized between two successive images, and wherein the weight 
w k m ( x ) is a function of the reliability of the motion vector (jc) . In this embodiment the 
interpolation (II) is carried out between two successive images F k and F k+1 which leads to a 
very accurate interpolated image F k+A . The implementation of the median-function in 

25 equation (II) is a further measure that reduces the risk of interpolating wrong flickering 
details. 

A further embodiment of the invention is characterized in that the reliability 

and thus the weight of the motion vector D* (3c) is a function of the difference between the 

luminous intensities I k ( x ) and I k+1 ( x+D„ (*) ) and is also a function of the relative frequency 

30 of occurrence of D k m (x) in the neighborhood of the location x in the image F k . 

In this manner the reliability of the motion vector comprises two components. The first 
component is the accuracy of the motion vector, which is established on the basis of at least 
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two luminous intensities of at least two successive images. The second comppnent is, the 
consistency, which is established on the basis of the relative occurrence of the motion vector 
in at least one image. 

In yet another embodiment the method of the invention is characterized in that 
the generation of interpolated luminous intensities of groups of pixels according to the 
method of the invention is only performed in those parts in the images of the data-signal 
where edges in the motion vector field are located. This leads to the advantage that the 
interpolation according to the invention is only performed in those parts of the image where 
the halo effect is most likely to occur. This can save processing time. 

A further embodiment of the invention is characterized in that the method 
comprises a step of edge detection, wherein an edge in the (motion vector field of) image F k 
is detected if at least one of the following inequalities (III) and (IV) is satisfied: 

II [3* (x - K)i - id; (x + k)\ || > t, an) 

|| [D k q (x - K)] 2 - [5; (x + K)] 2 1| > T, (IV) 

wherein q is a pre-determined integer value, where ||.|| is a function which yields the absolute 
value of its input argument and wherein [.] p is a function which yields the p* component of 
its vector input argument. Furthermore, in (III) and (IV), T is a pre-determined fixed real 
value threshold and K is a vector which is given by: 

k = (K x \K 2 )\ (V) 

where K\ and K { are integer values. In this embodiment of the invention, with (III) and (IV), 
the definition of an edge is determined by discontinuities that appear between motion vectors 
of the same kind in the same image. 

These and other aspects of the invention will be apparent from and elucidated 
with reference to the embodiments described hereinafter. 

In the accompanying drawings, in which certain modes of carrying out the 
present invention are shown for illustrative purposes: 
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, Fig. 1 is a schematic perspective overview of a data-signal comprising 
successive images; 

Fig. 2 shows an image of the data-signal; and 

Fig. 3 is a schematic diagram of a display apparatus comprising a device 
according to the invention for motion-compensated interpolation of the data-signal. 



In the motion compensating interpolation-method according to the invention a 
data-signal comprises a sequence of images F n , n=l,...,k,k+l,...,N. The data-signal can be a 
movie wherein each image is a picture. In Fig. 1 such a data-signal 2 is shown schematically 
by depicting the images F\ F k , F k+1 and F N . Furthermore an image F k+A is shown. This image, 
for example, can be obtained by interpolating the images F k and F k+1 . The real value A 
defines the relative position of the interpolated image F k+A in the sequence of images. 

Fig. 2 shows the image F k in more detail. Each image is built up of pixels. A 
pixel 4 is the smallest element of an image that can be given a certain luminous intensity. A 
number of pixels 4 constitute a group of pixels 6 (also called a block), this number of pixels 
in a group of pixels (or in a block) is pre-defined and can be any integer which is greater than 
zero. The location of a particular group of pixels in the image is indicated with a two- 
dimensional integer vector x = [x x , x 2 ] T , the components x\ and x 2 of x of the group of 
pixels 6 are shown in Fig. 2. Each group of pixels has a certain luminous intensity. The 
luminous intensity of the group of pixels at location x in the image F k is I k ( * ). For the 
interpolation of the data-signal 2 motion vectors D* (3c) 8 have to be established. These 
motion vectors are two-dimensional integer vectors that are normalized between successive 
images. A motion vector D* (x) is assigned to a group of pixels at the location x in the 
image F k and predicts the location of the corresponding luminous intensity in the image F k+1 . 
Since a multiple of M motion vectors D* (x) can be assigned to each group of pixels, the 
motion vectors are numbered m=l,2,...,M. 

In a first step, for a particular image, for each group of pixels of the image a 
motion vector 8 is calculated. Then, in a second step, each group of pixels will be assigned 
also those motion vectors that were already assigned to neighboring groups of pixels in the 
preceding step. In this example the group of pixels and its neighboring groups of pixels are 
all located in the same image. Furthermore, in this example, only the eight closest 
neighboring groups of pixels are used. In other embodiments of the invention the number and 




relative location of the neighboring groups of pixels can be chosen differently than i$ this 
example. 

At this stage each group of pixels has been assigned multiple motion vectors. 
In a third step the reliability of each motion vector is calculated. The reliability of a motion 
5 vector can be a function of the accuracy of the motion vector. The accuracy of the motion 
vector is determined by the difference of a predicted luminous intensity on the basis of the 
motion vector, wherein the motion vector is assigned to a first group of pixels of a first 
image, for a second group of pixels in a second image and the actual luminous intensity of 
the second group of pixels of the second image. The reliability of a motion vector can also be 
10 a function of the relative frequency of occurrence of the motion vector in the neighboring 

groups of pixels of the first group of pixels. Generally, there will be eight neighboring groups 
of pixels in the first image of the first group of pixels. The reliability of the motion vector can 
also be a function of both the accuracy of the motion vector and its relative occurrence. 

The fourth step comprises the interpolation between the images. This 
15 interpolation is in principle carried out for each motion vector that has been assigned to the 
groups of pixels. Each motion vector yields an interpolation result, wherein each 
interpolation result is provided with a weight that is calculated on the basis of the reliability 
of the motion vectors. 

Finally, in step five, the interpolation results are averaged in accordance with 
20 the weights, yielding the interpolation luminous intensities I k+A (jc ). All these interpolation 
luminous intensities together define the interpolated image F k+A 

The interpolation method as described hereinbefore is elaborated in more 
detail with the following mathematical formulas. For each of the M motion vectors of the 
corresponding group of pixels of the image F k an interpolation result i k+A m ( x ) for the image 
25 F k+A is calculated. The m* interpolation result i k+A m ( x ) corresponding to D* (x) is for 
example given by: 

i k+A m ( * ) = medianf (I k ( round {x - A * D * ( x) } ) , 

d\x)+I M (x))/2) , (VI) 
30 (l M (round{x + (1- A) * D* (3c)} )}, 

where medianf J is a function which gives the median value of its input arguments and 
roundf.J is a function which gives the nearest integer value to each component of its input 
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argument. The rationale of choosing a median function in (VI) is the following. If the 
luminous intensities l\round{x - A * Z>* (x) } ) and I k+l (round{x + (1 - A) * D* (x)} ) both 
correspond to a moving object, which object thus has to be visible in both the images F k and 
F k+1 , then the first and the third argument of (VI) will both lead to approximately the same 
5 values. Consequently a luminous intensity i k+A m ( * ), which is on ^of these two values, will be 
the desired output of (VI). However, if the moving object is not visible (for example since it 
is covered by the background of the picture in the image) in at least one of the images F k and 
F k+ \ then the first and third argument may lead to different values of the luminous intensity. 
In this case there is no consistency between the pictures of the two successive images and the 

10 best way to handle this situation is to prevent extreme luminous intensities. This can be done 
by choosing the median value of the two luminous intensities of the moving object in the 
images F k and F k+l (which are the first and third input argument in (VI)) and the average of 
the luminous intensities of the images F k and F k+1 of the group of pixels at the location x 
(this is the second input argument in (VI)). 

15 At this stage for each motion vector D k m (x) m=l,2,...,M an interpolation result 

i k+A m( * ) is obtained according to (VI)). Furthermore, a relative weight w k ro ( x Corresponding 
to the III th interpolation result i k+A m ( x )is calculated on the basis of the reliability of the motion 
vector. With this the resulting interpolated luminous intensity I k+A (* ) of the image F k+A can 
be calculated in the following way: 



20 



I k+A (x )=(Xm=l,..,M {W k m (ic ) * i k+A m (ic )}) / Em=l...,M {w k m (x )}, (VH) 



wherein Xm=i,..,M {•} is a summation from 1 to M over its argument {.}. 

In an advanced embodiment of the invention the reliability of the motion 

25 vector is a function of two components: its accuracy and its consistency. The accuracy of 
Z>* (3c) is a function of the difference between the luminous intensities I k ( x ) and 
I k+1 ( x + Z>* (3c) ). The consistency of D* (3c) is a function of the relative frequency of 
occurrence of D k m (3c) in the neighborhood of the location x in the image F k . By this 
approach the accuracy of the motion vector in relation to the size of the moving object, 

30 measured in groups of pixels, is estimated by comparing luminous intensities from at least 
two successive images. The consistency of the motion vector is estimated on the basis of 
information of at least one image. 



10 
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In a further advanced embodiment according to the invention the generation of 

interpolated luminous intensities according to the invention is only performed in those parts 

of the images of the data-signal where edges within the motion vector field are detected. 

Normally the interpolation of plane areas, i.e. areas without edges, in a motion vector field 

will not give rise to annoying interpolation errors, therefore in these parts of the images a 

conventional interpolation method can be used. 

In again a further advanced embodiment of the invention the method 

comprises a step of edge detection wherein an edge in the (motion vector field of) image F k is 

detected if at least one of the inequalities (VIII) and (IX) is satisfied: 

|| [D* (Jc - K)], - [D k q (x + K)] x || > T, (VIII) 
\\[D k g (x - K)] 2 - [5; (x + K)) 2 1| > T, (IX) 



15 wherein q is a pre-determined integer value and wherein ||.|| is a function which yields the 
absolute value of its input argument. Furthermore [.] is a function which yields the p* 

component of its vector input argument, in this example p can be 1 or 2. The real value T is a 
pre-determined threshold and K is a two-dimensional integer vector which is given by: 

20 K = (K X ;K 2 ) T , (X) 

where K\ and K\ are integer values. If more than one motion vector is assigned to a group of 
pixels (M>1) then a particular motion vector has to be chosen for that group of pixels by 
selecting an integer value for q (l<q<M) in (VIII) and (IX). For the detection of edges in the 
25 [1,0] T direction a possible procedure is to choose Ki unequal to zero and K2 equal to zero. 
Edges in the [0,1 ] T direction can be detected with Ki equal to zero and K2 unequal to zero. 

Fig. 3 schematically shows an embodiment of a device 10 for motion- 
compensated interpolation of a data-signal 2. The input 12 of the device is the data-signal 2 
30 and the output 14 provides interpolated luminous intensities for the image F k+A . The input 12 
is fed to a block 16 and a block 18. In block 16 the interpolation results according to (VI) and 
the interpolated luminous intensities according to (VII) are calculated. In block 18 a first 
motion vector for each group of pixels is calculated. Then, in block 18 a pre-determined 
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number of second motion vectors coming from surrounding groups of pixels are assigned to 
each group of pixels. This leads a total of M motion vectors per group of pixels. Next, each 
motion vector (x) is fed to its corresponding block 20.m, which block calculates the 
reliability of the motion vector. Then, by each block 20.m, m=l,...,M, the motion vector 
5 Z>* (3c) and its estimated reliability are fed to block 16. 

Furthermore it is an object of the invention to provide in a picture signal 
display apparatus that comprises means for receiving a data-signal 2, a device 10 for 
generating interpolated luminous intensities that includes means (16) for generating an 
interpolated image, and further means 22 for interleaving said data-signal with the at least 
10 one interpolated image and a display device D. Such a picture signal display apparatus for 
example can be used for generating a 100 Hz television signal out of a 50 Hz input signal. 

It will be clear to those skilled in the art that the invention can be practiced 
otherwise than as specifically illustrated and described without departing from its scope. For 
example, the described interpolation method can be based on images that are not successive. 
15 Furthermore it is possible that more than two images are used in the interpolation method. It 
is also an option to use the method and/or device of the invention for extrapolating instead of 
interpolating luminous intensities (in that case the real value A is greater than 1 or less than 
0). It should thus be noted that the above-mentioned embodiments illustrate rather than limit 
the invention, and that those skilled in the art will be able to design many alternative 
20 embodiments without departing from the scope of the appended claims. In the claims, any 
reference signs placed between parentheses shall not be construed as limiting the claim. The 
word "comprising" does not exclude the presence of elements or steps other than those listed 
in a claim. The word "a" or "an" preceding an element does not exclude the presence of a 
plurality of such elements. The invention can be implemented by means of hardware 
25 comprising several distinct elements, and by means of a suitably programmed computer. In 
the device claim enumerating several means, several of these means can be embodied by one 
and the same item of hardware. The mere fact that certain measures are recited in mutually 
different dependent claims does not indicate that a combination of these measures cannot be 
used to advantage. 

30 
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L A method of motion-compensated interpolation of a data-signal, which data- 

signal comprises successive images wherein each image comprises groups of pixels, the 
method comprising: 

generating (18) motion vectors, each motion vector corresponding to a group 
of pixels of one image, between a group of pixels of said one image and a second group of 
pixels of another image in the data-signal; 

generating (16) interpolated results as a function of these motion vectors; 

estimating (20) the reliability of each motion vector corresponding to a 
particular group of pixels; 

calculating (20) weights as a function of the reliability of the motion vectors; 

and 

generating (20) an interpolated luminous intensity of a group of pixels for an 
interpolated image by calculating, on the basis of these weights, a weighted average of the 
interpolated results. 

2. A method according to claim 1, wherein the interpolated luminous intensity of 

a group of pixels is calculated according to: 
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I k+A (ic ) = ffi m= l,,M {W* m (* ) * i* +A m (* )}) / 2m-l f .^ {W* m (* )}, 



(A) 



wherein I k+A ( x ) is the interpolated luminous intensity of the group of pixels of an 
interpolated image F k+A , wherein the location of the group of pixels in the image is defined by 
the integer two-dimensional vector x and where the real value A defines the place of the 
interpolated image F k+A in the image sequence F n , n=l,2,...,k,k+l,...,N, wherein Sm=i,..,M {•} 
is a summation from 1 to M over its argument {. } and where w k m ( x ) is a weight 
corresponding to the m m interpolation result i k+A m (x ): 



i k+A m (jc ) = medianf (I K ( round{x - A * D k m (*)} ) , 



(TU)+I k+l (* ))/2) , 



(B) 




(f + \round [ x + (1 - A) * £ * (3t) } V, 



wherein medianfj is a function which gives the median value of its input arguments and 
roundfj is a function which gives the nearest integer value to each component of its input 
5 argument, and wherein I k ( x ) is a luminous intensity of the group of pixels at location x of 
the image and wherein D k m (3c) is the 111 th two-dimensional integer motion vector, which is 
normalised between two successive images, of the M motion vectors which correspond to the 
group of pixels at location x and wherein the weight w k m ( x ) is a function of the reliability of 
the motion vector Z>* (3c) . 

10 

3. A method according to claim 2, wherein the reliability of the motion vector 
D* (3c) is a function of the difference between the luminous intensities I k ( x ) and 

I k+1 ( x + D k m (3c) ) and wherein the reliability is also a function of the relative frequency of 
occurrence of Z>* (3c) in the neighborhood of the location x in the image F k . 

15 

4. A method according to claim 1, wherein the generation of interpolated 
luminous intensities according to the invention is only performed in those parts of the images 
of the data-signal where edges in the motion vector field of the images are located. 

20 5. A method according to claim 4, wherein the method comprises a step of edge 

detection, wherein an edge in the motion vector field of image F k is detected if at least one of 
the inequalities (CI) and (C2) is satisfied: 

|| [D k g (3c - K)] x - [5; (x + K)] x || > T, (CI) 

25 

|| [5; (3c - K)] 2 - [5; (x + K)] 2 (I > T, (C2) 

1 

where q is a pre-determined integer value and wherein ||.|| is a function which yields the I 
absolute value of its input argument, [.] p is a function which yields the p* component of its j 

30 vector input argument, where T is a pre-determined fixed real value threshold and wherein K 
is a vector which is given with: 
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= jC,) r , (D) 

where JSTi and ATj are integer values. 

5 6. A device for motion-compensated interpolation of a data-signal, which data- 

signal comprises successive images wherein each image comprises groups of pixels, the 
device comprising: 

means (18) for generating motion vectors, each motion vector corresponding 
to a group of pixels of one image, between a group of pixels of said one image and a second 
10 group of pixels of another image in the data-signal; 

means (16) for generating interpolated results as a function of these motion 

vectors; 

means (20) for estimating the reliability of each motion vector corresponding 
to a particular group of pixels; 
15 means (20) for calculating weights as a function of the reliability of the motion 

vectors; and 

means (20) for generating interpolated luminous intensities of groups of pixels 
by calculating, on the basis of these weights, weighted averages of the interpolated results. 

20 7. A picture signal display apparatus, comprising: 

means (12) for receiving a data-signal, which data-signal comprises successive 
images wherein each image comprises groups of pixels; 

a device (10) for motion-compensated interpolation of said data-signal, as 
claimed in claim 6; 

25 means for generating at least one interpolated image on the basis of said 

interpolated luminous intensities; and 

means (D) for displaying the at least one interpolated image. 
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The invention relates to a motion-compensated interpolation of a data-signal, 
which data-signal comprises successive images wherein each image comprises groups of 
pixels, in which motion vectors are generated (18), each motion vector corresponding to a 
group of pixels of one image, between a group of pixels of said one image and a second 
group of pixels of another image in the data-signal, and interpolated results are obtained (16) 
as a function of these motion vectors. In accordance with the present invention, the reliability 
of each motion vector corresponding to a particular group of pixels is estimated (20), weights 
are calculated as a function of the reliability of the motion vectors, and interpolated luminous 
intensities of groups of pixels are generated for an interpolated image by calculating, on the 
basis of these weights, weighted averages of the interpolated results. 



(Fig. 3) 
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